1. Field of the Invention
The present invention relates to an image processing apparatus and, more particularly, to an image processing apparatus, an image processing method, and a program for detecting and correcting a defect pixel.
2. Description of the Related Art
In a recent digital image processing technique, a study field called a light field photography has remarkably been progressed. In the light field photography, first, an image pickup is performed so that a picked-up image including not only two-dimensional lighting information of a field of view but also ray directional information can be obtained by an image pickup apparatus including a photographing optical system such as lenses and the like and an image pickup element such as CCD, CMOS, or the like.
The picked-up image including the ray directional information corresponds to a picked-up image obtained with division into a number of pupils instead of division of a pair of pupils in what is called an auto focus detection (AF) of a phase-difference detection type. When such a picked-up image is directly observed, it (picked-up image) appears to be data which is not always information that is meaningfully arranged in meaningful order. However, by an image processing of a reconstruction type closely based on an image pickup step by which the picked-up image has been obtained, a refocus for newly focusing onto a desired object in the field of view after photographing can be performed. At present, the image pickup apparatus and image processing apparatus with such a construction have been proposed.
As an example of such image pickup apparatus and image processing apparatus, a Hand-held Plenoptic Camera (hereinbelow, referred to as a light field camera) disclosed in Non-Patent Literature 1 “Ren. Ng, et al., “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02” can be mentioned. The light field camera has an optical system for mainly forming an object image onto a “microlens” array having a predetermined pitch by a “main lens” serving as a photographing optical system in a normal image pickup apparatus. The light field camera also has an “image pickup element” for picking up the object image formed by such an optical system. The image pickup element is arranged behind the microlens array and has a plurality of photoelectric conversion elements of a pitch that is further smaller than the predetermined pitch.
In other words, the light field camera disclosed in the Non-Patent Literature 1 is a camera which can obtain an image having new information that is not provided in a well-known image pickup apparatus by a special image pickup step and an image processing which executes such a step as a prerequisite.
On the other hand, most of image processings which are executed by the normal image pickup apparatus are processings which are executed on the assumption that there is a continuity to a certain extent among signals of photoelectric conversion elements constructing the picked-up image. For example, there can be mentioned a real-time defect detecting method whereby each time the photographing is performed by the image pickup apparatus, a pixel is determined as a defect on the basis of level differences or the like between signals of that pixel and peripheral photoelectric conversion elements thereof. For example, a defect pixel correcting apparatus disclosed in Patent Literature 1 “the Official Gazette of Japanese Patent Application Laid-Open No. 2005-286825” has: a first defect pixel detection unit for detecting a defect pixel by comparing each of signals which are output from a plurality of pixels with a predetermined value; and a first correction unit for correcting a signal from the defect pixel detected by the first defect pixel detection unit. The defect pixel correcting apparatus further has a second defect pixel detection unit for detecting the defect pixel by comparing each of signals which are output from the plurality of pixels in which the correction to the defect pixel detected by the first defect pixel detection unit has been performed with a predetermined value. In such a construction, the defect pixel correcting apparatus of Patent Literature 1 is characterized by having a setting control unit for setting different values into the predetermined values of the first defect pixel detection unit and the second defect pixel detection unit.
According to Patent Literature 1, defect pixel information detected by the first defect pixel detection unit is stored, it is referred to by the second defect pixel detection unit, and whether the detected pixel is really a defect pixel or an object edge is discriminated. Patent Literature 1 also mentioned a point that when it is determined that the detected pixel is the object edge, the correction is not performed, so that an image quality obstruction can be prevented.
As will be also understood from the above description, a most important subject in the real-time defect detecting method is to discriminate whether the detected pixel is a defect pixel which should be a correction subject or the object edge.
A method whereby coordinates of a defect pixel which should be a correction subject are specified in, for example, a manufacturing step of an image pickup apparatus or an image pickup element and stored into a memory of the image pickup apparatus has also been proposed. An example of such an image processing apparatus is disclosed in Patent Literature 2 “the Official Gazette of Japanese Patent Application Laid-Open No. 2005-26794”. The image processing apparatus of Patent Literature 2 has: a storing unit for storing correction data including position information of a defect pixel of an image pickup element and information regarding its output level; and a correction processing unit for executing a correction processing of the defect pixel of an output signal from the image pickup element by using the correction data. The correction processing unit obtains a discrimination value by multiplying a threshold value which is determined in accordance with conditions which are classified in accordance with photographing conditions and a photographing environment with a conversion coefficient which changes in dependence on coordinates in a display screen of a subject pixel, and extracts the defect pixel serving as a correction subject from the correction data by using the discrimination value as a reference. Thus, the correction processing can be executed to the defect pixel.
Patent Literature 2 also mentions performing a correction corresponds to an accumulation time of photoelectric conversion charges which substantially differ in dependence on vertical scanning rows, which correction is peculiar to a CMOS type image pickup element of what is called a rolling read-out method. Further, Patent Literature 2 also mentions that excess and deficiency of the defect correction are eliminated by this correction and an increase of system resources such as a defect pixel storing memory and the like can be relatively avoided.
As will be also understood from the above, for the image pickup apparatus having the defect pixel storing memory, an obtainment of a balance between the number of defect coordinates to be stored and an effect of the defect pixel correction is also a problem to be solved.
Terminologies which are used in the following description will be defined here. In the light field camera, a unit pixel constructing “picked-up image (including ray directional information)” is called “signal of a photoelectric conversion element” or simply called “signal”, and a unit pixel constructing “reconstruction image” obtained by some reconstruction is called “pixel”.
In the conventional image pickup apparatus, there is no need to clearly distinguish such a definition that a unit pixel constructing the picked-up image is generally called “pixel”, a unit pixel constructing the image pickup element is called “signal of a photoelectric conversion element”, and a unit pixel constructing the final image obtained through signal processings is called “pixel”. However, for the light field camera, they are concepts which should be clearly distinguished from each other.
As mentioned above, in order to obtain a remarkable feature like a refocus reconstruction processing, for the light field camera, the special image pickup step occupies the important position as mentioned above. What the special image pickup step means is a simultaneous obtainment of the lighting information and the ray directional information. The ray directional information appears in intensity distribution of a number of photoelectric conversion elements corresponding to one of a plurality of microlenses constructing the microlens array. This provides such an image shift that the intensity distribution of the signals of a plurality of photoelectric conversion elements is deviated in accordance with a distance to the object and focusing of the photographing optical system. The image shift is such a phenomenon that the images constructed by the signals of the photoelectric conversion elements corresponding to the same quadrant of the microlens caused a deviation of coordinates, and this can be regarded as an expansion of a pair of image shifts in the phase-difference detection type AF to a number of image shifts.
That is, the picked-up image in which the signals of a number of photoelectric conversion elements are arranged in order of outputs from the image pickup elements without being subjected to some refocus reconstruction processing is data which is unsuitable to be directly observed.
Therefore, if the foregoing real-time defect detecting method which is performed on the assumption that there is continuity to a certain extent among the signals of the photoelectric conversion elements is merely applied to the picked-up image including the ray directional information, distinction of a defect pixel from the object edge, which is a problem to be solved is difficult.
Moreover, the light field camera has the special image pickup step as mentioned above and the photoelectric conversion elements of the pitch smaller than the pitch of the microlens array are disposed on the picked-up image. Therefore, even if a density of defects to be stored is same as that in the related art, the number of signals of the photoelectric conversion elements which should be correction subjects increases. Therefore, if it is intended to cope with it only by the storage of the correction subject coordinates as disclosed in Patent Literature 2, this results in a large increase of system resources such as a memory and the like. Therefore, it is difficult to avoid use or common-use of the real-time defect detection.
That is, when the above problems are summarized, in the case where the real-time defect detection and correction as mentioned above are performed to the light field camera, unless processing order of one type of or a plurality of types of defect detection and correction and the refocus reconstruction processing is accurately controlled, the following problems occur.
First, since the picked-up image includes the ray directional information, it includes the image shift. Therefore, in the picked-up image as it is, an image area in which it is difficult to distinguish the object edge and the defect exists.
Second, since the coordinates of the signals of the photoelectric conversion elements which contribute to the pixels on the reconstruction image are changed depending on the refocus reconstruction processing, the stored coordinate information of the defects cannot be directly applied.
Third, since the picked-up image is constructed by the signals of an extremely large number of photoelectric conversion elements, if the real-time defect detection is performed to all elements, it takes a long processing time, and on the contrary, if the coordinates of all of the defect elements are stored, system resources such as a memory and the like is increased.